An automated, reproducible way to get outside-eyes UX feedback: a panel of different vendors' vision LLMs each drive a real browser to organize a project, then give structured product feedback we can act on. It's the "measure vs Asana" design loop, automated.

How it works

• Panelists (via Bedrock Converse, AWS_PROFILE=personal, us-west-2 — no API keys, no SDK dep): Anthropic Claude Haiku 4.5, Amazon Nova Pro, Meta Llama 4 Maverick, Mistral Pixtral Large.
• Each panelist signs in and runs an agentic loop: screenshot + a numbered list of the page's interactive elements → the model picks one action (click/type/enter/goto/note/done by element number, not pixel coords — robust across vendors) → Playwright executes it → repeat (step cap 22).
• Then each answers a structured feedback questionnaire (delight & clarity 1–10, what worked, what was confusing, top improvements, vs-Asana).
• Output: per-step screenshots + transcripts + a synthesized report.md with a score table and improvement themes ranked by how many panelists raised each.

Run it

npm run dev -- --port 5173
AWS_PROFILE=personal npm run panel                          # wedding scenario, all 4
PANEL_SCENARIO=trip PANEL_ONLY=claude,nova npm run panel    # subset + other scenario
PANEL_BASE=https://taskflow.example npm run panel           # a live deployment

Scenarios: wedding (default — a couple planning their wedding, per the idea), launch, trip. Output → /tmp/tf-panel/<PANEL_RUN_ID>/.

First real signal (smoke run, Claude Haiku, wedding)

Delight 2/10 — "felt broken; no confirmation after creating a project; had to retry." Actionable backlog it surfaced: success toast / optimistic insert / loading state on project + task creation. (Partly the model wrestling with the create-form flow, partly a genuine lack of success feedback — worth fixing either way.)

Scope / gates

• Lives in scripts/llm-panel/ — a harness, exempt from the line/CRAP gate (not src/amplify); eslint doesn't lint those .mjs. No app code touched, so coverage/CRAP are unaffected.
• Documented in CLAUDE.md → Design → LLM feedback panel (incl. the inference-profile model-id gotcha).

Full quality gate green.